The Relaxed Hilberg Conjecture: A Review and New Experimental Support

The relaxed Hilberg conjecture states that the mutual information between two adjacent blocks of text in natural language grows as a power of the block length. The present paper reviews recent results concerning this conjecture. First, the relaxed Hilberg conjecture occurs when the texts repeatedly describe a random reality and Herdan’s law for facts repeatedly described in the texts is obeyed. Second, the relaxed Hilberg conjecture implies Herdan’s law for set phrases, which can be associated with the better known Herdan law for words. Third, the relaxed Hilberg conjecture is positively tested, using the Lempel-Ziv universal code, on a selection of texts in English, German, and French. Hence the relaxed Hilberg conjecture seems to be a likely and important hypothesis concerning natural language.